Method and system for creating and sending handwritten or handdrawn messages via mobile devices

ABSTRACT

A handwritten or handdrawing messaging system employs a handwriting messaging component operable with a messaging client of a mobile device connected to the data transmission network to set up a handwriting data capture area in the messaging client into which the user can enter handwritten or handdrawn input through a suitable manual input device, and then operates to capture the handwritten or handdrawn input and send it as a handwritten electronic message to the intended recipient. It also sets up a graphical data viewing area within the messaging client for viewing handwritten electronic messages sent to the user. Preferably, it is a small-footprint software module installed with the messaging client of the mobile device. The mobile device can be any type of wired or wirelessly-connected portable device operating on a computing platform, such as a personal digital assistant (PDA) device, game console or player device, or other mobile messaging device. The manual input device can be a touch-sensitive screen, tablet with pen input, stylus pad with stylus input, or an attached drawing pad. The system is particularly useful for messaging via mobile game player devices, so that people all over the world can send handwritten messages, chat, and play handdrawn games with each other.

This U.S. patent application claims the priority of U.S. patentapplication Ser. No. 10/436,722, filed on May 12, 2003, now issued asU.S. Pat. No. 6,763,373 on Jul. 13, 2004, which was acontinuation-in-part of U.S. patent application Ser. No. 09/978,472,filed on Oct. 15, 2001, entitled “Method and System for Creating andSending Handwritten Message”, issued as U.S. Pat. No. 6,564,249 on May13, 2003, which was a continuation-in-part of U.S. patent applicationSer. No. 09/687,351, filed on Oct. 11, 2000, entitled “Method and Systemfor Creating and Sending Graphical Email”, issued as U.S. Pat. No.6,304,898 on Oct. 16, 2001, which claimed the priority of U.S.Provisional Application 60/159,636 filed on Oct. 13, 1999, entitled“Graphical Email Drawing and File Attachment System”, all by the sameinventor.

FIELD OF THE INVENTION

This invention relates to the processing of handwritten input inelectronic messaging, and specifically to an electronic messaging systemfor sending and receiving handwritten or handdrawn messages via mobiledevices.

BACKGROUND OF THE INVENTION

Electronic messaging is a general method for sending and receivingcommunications as digital data between computers on a network. TheInternet has dramatically increased electronic messaging amongstmillions of users on global data networks. Many different forms ofelectronic messaging are being used to send and receive communicationsin a wide variety of forms of structured and unstructured data.Businesses make extensive use of electronic messaging to conductbusiness communications and transactions between trading partners.Electronic mail (or email) is a popular form of electronic messaging forcommunications between users. Typical email messages are composed oftyped text or a combination of typed text and text or graphical filesthat are attached to the email message and opened with the appropriateprocessor or viewer. As the popularity of the Internet continues to growworldwide, more and more people numbering in the billions are expectedto use email for communications.

Recent advances in technology and standards have expanded the types andforms of devices that can connect to the Internet. In addition todial-up and online connections between users computers and servers thatprovide information services and email services, many types of otherdevices are being connected to the Internet for communications purposes,including personal digital assistants (PDAs), text messaging pagers,digital cellphones enabled with Wireless Application Protocol (WAP),advanced digital game machines, digital set top boxes for televisions,and even CPU-controlled household appliances. Many of these deviceshaving Internet access do not require or are not adapted to use akeyboard for inputting data. While there are other types of inputdevices that enable handwritten or handdrawn input, such as touchsensitive screens, stylus pads, optical pens, etc., they have not beenenabled for electronic messaging and other communication functions.

Handwritten or handdrawn input can be more convenient to use than akeyboard and, in many situations, would be uniquely necessary forcertain types of communication. Many written language systems, such asJapanese, Korean, Chinese, Arabic, That, Sanskrit, etc., use cursive orideographic characters that are very difficult to input by an equivalentmethod via keyboard. For example, text input of the Japanese writtenlanguage requires the use of simulated phonetic spelling methods(romanji, hiragana, and/or katakana) to select from thousands ofpossible kanji characters. Many mobile devices such as PDAs do not havekeyboards due to their limited size and form, or would become cumbersometo use if a keyboard must be attached or if text must be entered bycursoring through displays of softkeys. Disabled or hospitalized peoplewho have limited hand mobility may not be able to use a keyboardeffectively. Current legal and financial institutions still rely heavilyon the use of handwritten signatures to validate a person's uniqueidentity. And in many instances, people find it much easier tocommunicate an idea by drawing a picture, or prefer handwriting ordrawing a picture as more personal or expressive communication thantyping text on a keyboard.

There is thus a clear need for an electronic messaging system thatallows people to communicate with their own handwriting or drawing, ascontrasted to typed text. This need will continue to grow as the numbersof global users and Internet-connected devices increase. None of thecurrent electronic messaging methods allow a user to compose,manipulate, store, send, receive, and view a handwritten or handdrawnemail message.

SUMMARY OF THE INVENTION

In accordance with the present invention, an electronic messagingsystem, and related method, comprises a server component operable on aserver computer connected to a data transmission network for receiving ahandwritten message in electronic format sent from a user and deliveringit to a recipient to whom it is addressed, and a handwriting messagingcomponent operable with a messaging client of a mobile device having aconnection to the data transmission network, wherein said handwritingmessaging component sets up a handwriting data capture area within saidmessaging client into which the user can enter handwritten or handdrawninput through a suitable manual input device, and said handwriting datacapture area operates to capture the handwritten or handdrawn input asgraphical data and send it as a message in electronic format on the datatransmission network to the server component for handling and deliveryof the handwritten electronic message to the addressed recipient. Thehandwriting messaging component is also operable for setting up agraphical data viewing area within the messaging client of the mobiledevice for viewing the graphical data sent as a handwritten electronicmessage to the user.

In a preferred embodiment, the handwriting messaging component is asmall-footprint software module installed with the messaging client ofthe mobile device. The mobile device can be any type of wired orwirelessly-connected portable device operating on a computing platform,such as a personal digital assistant (PDA) device, game console orplayer device, or messaging device. The manual input device can be atouch-sensitive screen, tablet device with pen input, stylus pad withstylus input, or an attached drawing pad. Preferably, the handwritingmessaging component sets up the handwriting data capture area as adrawing editor/viewer on a touch-sensitive display screen of the mobiledevice that allows the user to compose, manipulate, and view handwrittenor handdrawn messages by directly writing thereon. The editor/viewer caninclude standard drawing tools such as those for drawing line size,color, background image as a markup image, wallpapers, stationarystyles, border selections, circle and polygon shapes, paintbrush, spraypaint, flood fill, color palette, undo, scrolling, and/or otherdrawing/editor functions.

Other objects, features, and advantages of the present invention will bedescribed in further detail below, with reference to the followingdrawings:

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1A is a diagram illustrating the connection of client computerswith pen devices to an email server computer on the Internet, and FIG.1B shows the email handling components of the client and servercomputers;

FIG. 2A is a process flow diagram of an SMTP server version of thegraphical email system, and FIG. 2B is a schematic illustration of thenetwork connections of this version of the graphical email system;

FIG. 3A is a process flow diagram of a Lotus™ Domino server version ofthe graphical email system, and FIG. 3B is a schematic illustration ofthe network connections of this version of the graphical email system;

FIG. 4A is a process flow diagram of a real-time messaging serverversion of the graphical email system, and FIG. 4B is a schematicillustration of the network connections of this version of the graphicalemail system;

FIG. 5 is a schematic illustration of the network connections of anInternet email server version of the graphical email system;

FIG. 6 is a schematic illustration of the network connections of aWAP-enabled cellphone or PDA version of the graphical email system; and

FIG. 7A is a schematic illustration of the user interface for thegraphical data capture and drawing functions of the graphical emailsystem; and FIG. 7B is an example of an editor/viewer interface for thegraphical email system on a personal digital assistant (PDA).

FIG. 8 is a schematic illustration of the handwritten messaging systemoperable on a wireless or cellphone device network.

FIG. 9 is a block diagram illustrating the operational sequence for thehandwritten messaging system.

FIG. 10 is a schematic diagram of the interconnection of components ofthe handwritten messaging system.

FIG. 11 illustrates the use of the handwriting messaging component forenabling the messaging client of mobile devices to send and receivehandwritten messages via a data transmission network.

DETAILED DESCRIPTION OF INVENTION

In the following description, exemplary embodiments of the handwrittenor handdrawn messaging system of the present invention are provided fordifferent network environments, such as handwritten email messaging onthe Internet, real-time (instant IM) handwritten messaging on IMnetworks, handwritten messaging on wireless or digital cellphonenetworks, and handwritten messaging by mobile devices on wireless datanetworks. This U.S. patent application is directed particularly tohandwritten messaging by mobile devices on wireless data networks.

Referring to FIG. 1A, the general system architecture of the handwrittenmessaging system (and related method) is illustrated for email networks.A plurality of client computers 110, 111, etc., adapted for handwritingor handdrawing input are used by users to connect via a network (theInternet or any type of multi-user network) to a server computer 120. Inthe context of this description, the term “computer” is used to refer toany type of data processing device which is capable of executingdigitally programmed instructions. The term “client computer” refers toany computer capable of connecting to a server computer on a network andperforming a function with the server computer. An example of a typicalcomputer for use on a network is a Pentium or higher class PC withWindows operating system of Microsoft Corp., Redmond, Wash., orMacIntosh or higher class PC with Macintosh OS operating system of AppleComputer Corp., Cupertino, Calif. However, a client computer can also bea wide range of other network-connectable computing devices such aspersonal digital assistants (PDAs), text messaging pagers, digitalcellphones enabled with Wireless Application Protocol (WAP), advanceddigital game machines, digital set top boxes for televisions, and evenCPU-controlled household appliances. The term “server computer” is usedto refer to any type of data processing device that is connected to anode on a network for providing services to client devices on thenetwork.

The client computer 110 is what a sender or recipient uses to composeand send, and receive and view, handwritten or handdrawn email messages.The client computer preferably is of the type that can run a standardweb browser which supports an email client, such as those compatiblewith Microsoft IE 4.x browsers, licensed by Microsoft Corp., ofBellevue, Wash., or Netscape 4.x web browsers, licensed by AmericaOnline, Inc., of Fairfax, Va. The standard browsers preferably alsosupport the Java Virtual Machine, Version 1.2 or higher, licensed by SunMicrosystems, of Mountain View, Calif., which is the preferred platformfor programming and distributing the handwriting messaging software inthe present invention as Java applets or Java-based plug-ins to thebrowsers. The standard web browsers connect to the network using anystandard protocol recognized on the network. For example, on theInternet standard web browsers use the TCP/IP protocol. Connection tothe network may be made by modem over a dial-up telephone line, DSLline, cable modem, Internet access connection, or a local area network.

The handwriting or handdrawing input device can be a touch screen, peninput device, stylus pad, optical pointer, mouse, etc., which isattached to the client computer to allow the user to handwrite orhanddraw messages in a graphical data capture area of the email clientof the web browser set up for that purpose. Examples of such inputdevices include the Wacom Graphire™ pen tablet sold by Wacom TechnologyCorporation, of Seattle, Wash., which attaches to the client computervia a serial or USB port. The pen device could also be integrated with atouch screen, e.g., as also offered by Wacom, or part of the computeritself, e.g., as offered with the Sharp SJ5, Copernicus and Pro Stationssold by Sharp Corporation, of Tokyo, Japan.

The server computer 120 is a central processing server for the graphicalemail system. It is connected to the network to communicate with theclient computers using, for example, the TCP/IP protocol on theInternet. In the preferred implementation, the server computer storesthe graphical email handling software that is downloaded to the clientcomputers. It authenticates users against a directory of authorizedusers and manages and tracks the state of concurrent user sessions. Forsending and receiving graphical email, it communicates with thegraphical email software on the client computers. When a handwritingmessage is composed on the client computer, the server computer receivesthe graphical email message and stores it in a database. When ahandwriting message is to be viewed by the client computer, the servercomputer fetches the graphical email message from the database and sendsthe message to the client computer for display as a handwriting image.

Referring to FIG. 1B, the graphical email handling components of thepreferred system are illustrated. The client computer 210 includes aHandwriting Messaging Client 211 which handles sending and receivinggraphical email messages, a Java Virtual Machine (VM) 212 which sets upthe graphical data capture area and display area for the handwritingmessage, and the Web Browser 213 which provides the user interface forconnecting to the Internet (or other network). There are two versions ofthe Handwriting Messaging Client software described below, i.e., a Javaapplet and a Java application version. The Java applet version is usedfor sending and receiving handwritten email messages via a servercomputer's mail server functions. The Java application version of thehandwriting messaging client software is installed with the users'browsers for real-time messaging between users or via a real-time Javaserver. The client software includes a drawing editor/viewer forcomposing and viewing handwritten email messages, as well as thefunctions to communicate with a server in a server-client configuration.

The server computer 220 includes an HTTP Server Interface 221 forconnection to the Internet, a User Directory 222 for identifying emailaddresses of authorized users, User Message Queues 223 for deliveringreceived messages to users, a Downloadable Software Server 224 fordownloading the graphical email software to client computers, a MailServer Interface 225 for handling sent and received email, a JavaVirtual Machine (VM) 226 which provides the platform for interactingwith the users' graphical email software, and a Handwriting MessagingServer 227 which formats email messages using the graphical datacaptured by the software on the client computers. There are fourversions of the Handwriting Messaging Server described below, i.e., aHandwriting Java Server Version, a Domino Server Version, a Real-TimeJava Server Version, and an Internet Email Server Version.

Handwriting Java Client and Handwriting Java Server Version

With reference to FIG. 2B, this version of the system uses bothhandwriting client and server software components along with the usualemail server functions. A Handwriting Java Client 210 a operates in aweb browser (such as Netscape Navigator or Microsoft Internet Explorer)as a Java applet on the client computer 210. The applet 210 a provides adrawing editor to compose and send handwritten email messages, and onthe receiving end, also provides the drawing viewer to view thehandwritten message. The implementation of a Java applet for thehandwriting messaging function is described in further detail below.Generally, the use of a Java applet to provide a drawing applicationintended to run in a browser is known to those knowledgeable in thisfield. As one example, a Java applet used to set up a drawing editor ina browser for a “whiteboard” application run on an intranet is theSameTime™ whiteboard applet offered with Lotus™ Notes, a workgroupsoftware suite distributed by Lotus Development Corp., of Cambridge,Mass., which is a division of IBM Corp., Armonk, N.Y.

A Java applet is created specifically for setting up a drawingeditor/viewer operable with an email client running with a web browserfor capturing, sending, receiving and viewing a handwriting orhanddrawing email message. The Java applet consists of a set of moduleswhich run the different messaging functions of the drawingeditor/viewer. The data capture and sending module operates by firstrecognizing the handwriting input device connected to the clientcomputer, then creating a temporary memory space for holding the inputreceived from the input device, capturing the input signals from theinput device (usually a series of coordinate data inputs representingtime-sampled coordinate positions detected for the trace of the inputdevice), converting the input signals to pixel data (screen coordinates,color, intensity), and displaying the pixel data in a display area or“panel” of the email client to show the handwriting or handdrawn imageinput by the user through the input device. The display allows the usersending a handwritten email message to verify what is being written, andis also the same display that allows a user receiving the handwrittenemail message to view the corresponding image. An example of a datacapture and sending module according to the invention is illustrated inthe source code listing appended hereto as Appendix I.

When the graphical email message is completed, the user addresses themessage to a recipient and sends it. The Handwriting Java Client formatsand sends the email message with the pixel data to the Handwriting JavaServer component 220 a on the server computer 220, which converts thepixel data to a GIF file attachment to a standard email body. TheHandwriting Java Server component 220 a communicates with an SMTP emailgateway computer 240 to send email messages using the industry-standardSMTP Internet protocol. The SMTP email gateway sends the email messagesto mail servers 250, such as an industry-standard IMAP (Internet MessageAccess Protocol) mail servers like MS Exchange or Lotus Domino on theInternet. Email recipients can retrieve their email from the mailservers 250 using a standard Internet email client 260, such asMicrosoft Outlook, Pegasus Mail, Eudora mail, or Lotus Notes. When thegraphical email is retrieved with a standard Internet email client, thehandwritten drawing is viewed as a file attachment using a GIF vieweroperates with the web browser. Email recipients on client computers 230who have the Handwriting Java Client 210 a in their web browser canreceive their handwritten email messages directly. The graphical emailmessage is retrieved from the Handwriting Java Server component 220 a onthe server computer 220 and displayed in the Handwriting Java Clientviewer as a handwritten or handdrawn image.

With reference to FIG. 2A, the specific process steps involved withsending a handwritten email message and viewing it by the recipient aredescribed in detail below:

1. The Handwriting Java Client software is downloaded from theHandwriting Java Server to the client computer through a web page thatis displayed in a web browser.

2. The Handwriting Java Client software is initialized and establishes aconnection to the Handwriting Java Server using the industry-standardTCP/IP and remote method invocation (RMI) protocols. Afterinitialization is complete, the Handwriting Java Client softwaredisplays a drawing composition editor that is used to compose thehandwritten message.3. The handwritten message is composed by the user in a graphical datacapture area set up by the drawing editor, selecting the appropriatewriting and drawing tools, colors, and styles as offered in theHandwriting Java Client software.4. While the user is drawing in the graphical data capture area, thepixel data representing the drawing is stored in local memory. When thegraphical email message is completed, the user addresses the message toa recipient using Javascript fields on the web page in which the Javahandwriting client is embedded.5. When the user issues a “Send” command, the Handwriting Java Clientformats the message and sends the pixel data to the Handwriting JavaServer. The graphical message is still in GIF format at this time.6. The Handwriting Java Server processes the graphical message datausing standard base64 encoding. This turns the data into ASCII text thatcan be transmitted as standard email data packets by the HandwritingJava Server.7. The Handwriting Java Server creates an outgoing email message thatcontains the encoded handwritten message as a GIF attachment.8. The Handwriting Java Server sends the outgoing email message with GIFattachment via the SMTP (Simple Mail Transfer Protocol) gateway 240.9. The SMTP gateway transfers the message to an IMAP mail server basedon the recipient's address. The IMAP server allows clients running it toretrieve mail from the host mail server also running the protocol,similar to POP-3, but with extended capabilities. Recipients can openthe email as a standard email message with a GIF attachment (steps 10a,10b below) or with a Handwriting Java Client applet if downloaded totheir web browser (steps 11a, 11b, 11c below).10a. The IMAP server sends the email with the handwritten message as anattached encoded GIF file to an external email address for therecipient.10b. When the recipient opens the email containing the attachment, themessage can be displayed on their computer using a GIF viewer.11a. The IMAP server sends the email with the handwritten message as anattached encoded GIF file to an internal email address, i.e., to anaddress on a server computer 220 that is running the Handwriting JavaServer component 220 a.11b. The Handwriting Java Server decodes the attached GIF file intopixel data and sends it to the Handwriting Java Client applet running inthe recipient's web browser.11c. The Handwriting Java Client receives the pixel data from theHandwriting Java Server and renders the pixel data as a handwritten orhanddrawn image in the drawing editor/viewer.Handwriting Java Client and Domino Server Version

With reference to FIG. 3B, this version of the system uses theHandwriting Java Client 310 a along with a Lotus™ Domino Server on theserver computer 320. As before, the Handwriting Java Client 310 aoperates in a web browser as a Java applet on the client computers 310,330. The applet 310 a provides a drawing editor/viewer to compose andview handwritten messages. For sending and receiving email internally,the applet 310 a communicates with the Domino Server on the servercomputer 320. The Domino server sends email messages among users withclient computers running Java client software connected to the server'sintranet. The Domino Server communicates with an SMTP email gatewaycomputer 340 to send email messages externally using the SMTP Internetprotocol. The SMTP email gateway sends the email messages to mailservers 350 on the Internet. External email recipients can retrievetheir graphical email using a standard Internet email client 360. Thehandwritten drawing can then be viewed as a file attachment using a GIFviewer such as a web browser.

With reference to FIG. 3A, the process steps involved with sending ahandwritten email message through a Domino Server and viewing it by therecipient are described in detail below:

1. The Handwriting Java Client software is downloaded from theHandwriting Java Server to the client computer through a web page thatis displayed in a web browser.

2. The Handwriting Java Client software is initialized and establishes aconnection to the handwriting Domino Server using the TCP/IP and remotemethod invocation (RMI) protocols. After initialization is complete, theHandwriting Java Client software displays a composition editor that isused to compose the handwritten message.3. The handwritten message is composed by the user selecting theappropriate writing and drawing tools, colors, and styles.4. While the user is drawing, the pixel data representing the drawing iscaptured in the input data area and stored in local memory. The useraddresses the email message to a recipient using Javascript fields onthe web page in which the Handwriting Java Client is embedded.5. When the “Send” command is issued, the Handwriting Java Clientencodes the pixel data in base64 format.6. The Web page along with the encoded image is posted to the DominoServer using Javascript.7. An agent on the Domino Server decodes the image and creates an emailmessage with an attached GIF file.8. The Domino server sends the email message with attachment to anexternal recipient's address via an SMTP (Simple Mail Transfer Protocol)gateway.9. The SMTP gateway transfers the email message to a mail server, whichroutes the message to the recipient's email box.10a. an external recipient's e-mail client retrieves the email messagewith the GIF attachment from an Internet email server.10b. When the user (recipient of the email) opens the email containingthe GIF attachment, the handwritten message in the attachment can bedisplayed using a GIF viewer.11a. If the recipient has an internal email address handled by a DominoServer, the server retrieves the email message with the GIF attachment.11b. The Domino Server sends the email message to the client computer asa web page when the client requests the page via their web browser emailclient.11c. The client's web browser email client displays the handwrittenmessage as an image.Handwriting Java Client and Real Time Server Version

With reference to FIG. 4B, this version of the system uses theHandwriting Java Client with a Real Time Handwriting Java Server. Asbefore, the Handwriting Java Client 410 a operates in a web browser as aJava applet on the client computer 410, 430. The applet 410 a provides adrawing editor/viewer to compose and view handwritten messages. Theapplet 410 a communicates with the Real Time Java Server component 420 aon a server computer 420. The receiving Handwriting Java Client isnotified when an email message for that user has been sent to the RealTime Java Server, and retrieves the email message from the server. Inthis way, communication can take place using the Handwriting Java Clientin near real-time. This version is useful for users using mobilecommunication devices, such as PDAs, WAP-phones, etc.

With reference to FIG. 4A, the process steps involved with sending ahandwritten message through a Real-Time Handwriting Java Server andviewing it by the recipient are described in detail below:

1. The client computer has a downloaded version of the Handwriting JavaClient software, and establishes a connection with the server computeron which the Real Time Java Handwriting Server is running.

2. The client computer identifies the user to the server by entering auser name and password.

3. The server maintains a list of registered users, and a list ofcurrently active users. When the client logs onto the server, the serverlooks up the user name and password in order to authenticate that personas a registered user, then the user is added to the list of activeusers.4. The client computer displays a list of active and inactive userswhich is downloaded from the server.5. The user clicks on a name from the list of active users to identifythe user to whom they want to send a message.6. After the user selects a name from the list of active users, thehandwriting editor is displayed in the Handwriting Java Client.7. The user creates a message by selecting the appropriate writing anddrawing tools, colors, and styles.8. The Handwriting Java Client formats the message and stores it aspixel data until it is ready to be sent.9. The completed message is sent to the Real Time Java HandwritingServer in pixel format.10. The server receives the message and puts it into a repository whereit can be retrieved by the user to whom it is addressed.11. All active client computers poll the repository on the server everyfive seconds to see if there are any messages for them.12. When the Handwriting Java Client on the recipient's computerdiscovers a message in the repository, the client computer requests themessage from the server.13. The server retrieves the message from the repository and sends it tothe client computer of the recipient.14. The recipient's client computer displays the message in theHandwriting Java Client's drawing editor/viewer.Handwriting Java Client and Internet Email Server Version

With reference to FIG. 5, this version of the system uses theHandwriting Java Client along with a standard Internet email mailserver. In this version, the Handwriting Java Client is installed as aplug-in to the client computer's web browser and operates as describedbefore, and there is no Handwriting Java Server component. TheHandwriting Java Client 510 a operates in a web browser as an installedJava applet on the client computer 510. The applet 510 a provides adrawing editor/viewer to compose and view handwritten messages. TheHandwriting Java Client formats the message and stores it as pixel datauntil it is ready to be sent. The message is addressed using Javascriptfields on the Web browser form in which the Java applet is embedded. Thepixel image is converted into a GIF file and attached to the emailmessage. The Java applet 510 a communicates with the Mail Servercomputer 540 either directly or through an SMTP gateway computer 520.The Mail Server 540 sends the email message with encapsulated GIF imagedirectly to the recipient's client computer 530, and the recipient viewsthe attached GIF file using whatever GIF viewer they have available ontheir computer.

Handwriting Java Client and Wireless Internet Email Server Version

With reference to FIG. 6, this version of the system uses theHandwriting Java Client along with a standard Internet email mail serverproviding email service to wireless client computers, such as WAP-phonesand PDAs. As before, the Handwriting Java Client 610 a operates in a webbrowser as an installed Java applet on the client computer 610. Theemail message is formatted with the handwritten image converted into anattached GIF file. The Java applet 610 a communicates with the MailServer computer 640 either directly or through an SMTP gateway computer620. The Mail Server 640 includes a Wireless Application Protocol (WAP)interface which sends the email message with encapsulated GIF imagethrough a Wireless Service Provider having WAP handling capability tothe recipient.

The recipient's computer can be a thin client 630 a such as a digitalcellphone with a WAP interface for receiving the email message viaInternet and displaying the GIF file via its mini-web browser.Alternatively, the client computer may be a more robust, mobile clientcomputer 630 b such as a Palm Pilot™ or similar type of PDA, which hasthe memory and CPU capacity to have a Handwriting Java Client installedwith its web browser and use it for composing and sending handwritingemail messages as well as viewing them. The mobile client computer canthen use the Handwriting Java Client to format the handwritten messageas an attached GIF file (described in the Internet Email Server version)or as pixel data sent with the email for viewing by another clientcomputer having a Handwriting Java Client running in its web browser (asdescribed in the Real-Time Server version).

A handwriting messaging application written for a palm-top or PDA devicewould currently have to be written in C or C++ because there are nocurrent Java Virtual Machine adaptations that can be used for suchdevices. However, several efforts are underway to create such Java VMmodules for palm-top and PDA devices. Using C/C++ to write fullapplications on palm-top devices has the current advantage that thesecurity sandbox imposed on Java applets does not exist, therebyallowing a wider variety of messaging applications to be written, aslong as actual implementations are kept simple (due to low CPU power andmemory storage availability). The handwriting client can be kept simpleby including only pen and eraser tools and different line thicknesses.Only palm-top devices with color displays would need a color palette;black and white palm-top devices would dither incoming messages to blackand white while sending only two-color images. Since the client iswritten in C or C++, networking would be limited to standard TCP/IPcommunications instead of Java's RMI. For communications through aproxy, packets can be wrapped in HTTP and sent through an HTTP proxy. Ifa handwriting server component is used with the palm-top devices, itwould remain largely the same as described above, except that it wouldhandle only standard TCP/IP communications, and would add the ability toreceive messaging information wrapped in HTTP packets from behind afirewall.

The functions of the module of the client component which converts andsends the handwritten message for handling as an email message dependsupon the configuration of the system. If the system is configured with aLotus Domino server on a Java VM platform, then the captured handwritingdata is sent as a message to the Domino server in the form of a streamof pixel data which is converted by the server to a Graphics InterfaceFormat (GIF) file and appended with an email header to form an emailmessage handled by the server's email service. If the system isconfigured with a standard type of Internet email server, then thecaptured handwriting data can be converted at the client computer to aGIF file and sent as an email message to the Internet email server. Ifthe system is configured for real time messaging between client devices,the client device can send the handwritten message as a GIF fileappended to standard email, or send it as pixel data to a real-timemessaging server which provides real-time messaging service betweenclient devices.

In FIG. 7A, the user interface for the drawing editor/viewer of theHandwriting Java Client is illustrated. For composing a handwrittenmessage, the editor/viewer 710 has a graphical data capture area 720 inwhich handwritten input can be entered with a touch pen or on a styluspad and captured as pixel data. The editor/viewer 710 can offer a suiteof standard types of drawing tools such as making line, circle, orpolygon shapes and spraying and filling image areas. The graphical datacapture area 720 is also the graphical image viewing area for emailreceived with an attached GIF file or pixel data. In FIG. 7B, a typicallayout for a PDA of the editor/viewer interface with drawing tools andcolor palette is illustrated.

Handwriting Client and Wireless Real Time Server Version

The version of the system illustrated in FIG. 6 may be modified forreal-time messaging between PDAs and mobile communication devices of the“always on” type that are connected to a real-time server through anInternet Wireless Service Provider, rather than a WAP-type phoneinterface. Examples of “always-on” types of Internet wireless messaginginclude Research in Motion's (RIM's) BlackBerry™ pager email devices,Motorola Timeport™ P935 Personal Communicator, or the Palm's announcedi705 wireless device with always-on access to email.

Similar to the sequence outlined above for the Handwriting Java Clientand Real Time Server Version, the always-on wireless messaging device isprovided with a handwriting client that sets up a handwriting (e.g.,stylus) data capture area in the device to capture the sender'shandwriting input and send it as an attachment with a wireless email ormessage transmission. The wireless service provider employs a real-timeserver that receives the wireless email or message through theappropriate wireless Internet service interface and routes the messageto a private mail or message repository for the intended recipient. Ifthe recipient has an always-on device, the message in that person'sprivate repository is sent to the recipient's device through wirelesstransmission, where it pops up as a message in the device display. Witha handwriting viewer or a mode for viewing attached graphics files, therecipient sees the message instantaneously as handwriting.

The wireless real-time server version allows people to communicateprivately and personally by instantly sending handwriting and othergraphical data to each other. The handwriting client has utilities toallow a user to maintain an address book of people they wish tocommunicate with. When the user logs on to the real time server, theyare authenticated with a user name and password and are registered as anactive user. The handwriting client allows the user to see other usersthat are currently on-line and to initiate a private, real timecommunication session with an on-line user. The handwriting clientaccepts handwritten and other graphical messages, packages the data in amessage and sends it to the recipient. The recipient receives an audiblenotification that they have a message and is able to view the message intheir handwriting client. The recipient is also able to reply to thesender by composing a new handwriting message or copying and writingover the original message that was received. This person-to-personcommunication can continue back and forth, effectively allowing theparties to communicate using handwriting and drawings.

This method of instantly sending handwriting between to parties differfrom other real time communication systems such as chat sessions andsystems commonly referred to as “electronic whiteboards” and “instantmessaging systems” in the following ways.

1. Private message pools for graphical messages: In the invention,graphical messages to be sent to a recipient are stored in therecipient's private message pool and not a shared, common pool. Therecipient's client software polls their personal pool and not a commonarea. This method supports private, one-on-one communications similar toemail, except in real time. Chat rooms and whiteboards are intended forgroup communication and incorporate shared message pools.2. Graphical handwriting client: The invention allows users tocommunicate using handwriting input rather than text input. Thehandwriting client is essentially a graphics input editor implemented inJava which communicates with a real time server. The real time serverconverts the input data into a real time messaging format that is storedin a repository location and retrieved by the recipient's client.3. Control over when message is sent: In the invention handwritingmessages are composed privately in the handwriting client and sent ondemand by the sender, similar to how standard text email is composedlocally and sent on demand. Whiteboard systems send graphical data inreal time as they are drawn to all users that are viewing thewhiteboard.

The wireless real time server version may also be implemented onportable PDA devices and mobile phones that use the Short MessagingService (SMS) or Enhanced Messaging Service (EMS) for wireless instantmessaging on wireless phone service. SMS and EMS now permit graphicsattachments to be sent with messages. The handwriting client softwarecan be implemented to run on these mobile devices using a micro orpersonal Java platform that are becoming available on these devices. Thereal-time server can correspondingly be adapted with an interface tosupport handwriting messages sent through the messaging formats used bythese mobile messaging devices.

Handwriting Java Client for MMS Messaging in Cellphone Network Version

In FIG. 8, the MMS Cellphone Messaging version of the handwritingmessaging system allows people to communicate privately between MMScapable terminals by composing and sending handwriting and othergraphical data to each other. The originator composes a handwritingmessage using a pen device connected to the handset and addresses themessage to the receiver. An MMS compliant message is created and sent toan MMS Center (MMSC) that is provided by the originator's wirelessservice provider. The MMSC attempts to forward the message to thereceiver. If the receiver is unavailable, the MMSC stores the messageand delivers the message later. If the message cannot be deliveredwithin a certain amount of time, the message is discarded.

MMS is becoming a standard method for sending and receiving multimediacontent over wireless networks. Just as our original invention tookadvantage of standard methods for sending SMTP based email betweencomputers, the same principle can be implemented for digital wirelessphone devices and Internet-capable phones (I-phones) using a digitalmessaging protocol, such as the now standard MMS protocol (as publishedby the MMS Conformance Group), for wireless mobile devices. Thehandwriting client software can readily be implemented to run on mobiledevices using the Java 2 Micro Edition (J2ME) platform that are becomingcommonly available on these devices.

The Java client software is programmed to set up the graphical datacapture input area in the client device and capture the graphical(pixel) data for conversion to a standard format such as jpeg or gifhandled by the digital messaging protocol used in digital phone devicesor I-phones. For example, the MMS protocol can support jpeg and giffiles converted by the handwriting client application in the followingmanner. While the user is drawing with a stylus or pen on the graphicaldata capture area, the pixel data representing the drawing are stored inlocal memory in the client device. The user can address the handwrittenmessage to a recipient using Javascript fields of the graphicalinterface in which the Java handwriting client is embedded. When theSEND MESSAGE command is selected by the user, the Handwriting JavaClient formats a message with the pixel data in gif format to theHandwriting Java Server, which then processes the message and sends itto the MMS-compliant client device of the recipient using the MMSCserver protocol.

In FIG. 9, the steps involved with sending a handwritten message andviewing it by the recipient are illustrated using MMS-compliant mobilephone devices.

1. The handwriting client software is used to compose a handwrittenmessage. The handwriting client software is running in a J2MEenvironment that is available on mobile devices. The mobile phone alsohas a pen enable screen or an attached pen pad device that captures theuser's pointer movements. The handwriting client software allows theusers to select drawing tools and captures the handwriting image createdby the pointer movements. Images can be captured with the mobile devicescamera or downloaded as a background graphic that can be annotated withhandwriting and is included in the message.2. The originator addresses the message to the recipient(s).3. The originator's terminal contains information about the wirelessprovider's MMSC, initiates a Wireless Access Protocol (WAP) connectionand sends the handwriting message as the content of the WSP POST command4. The MMSC accepts the message and responds to the originator over thesame WAP connection. A “message send” indicator is displayed on theoriginator's terminal.5. The MMSC uses WAP PUSH command to attempt to send an indicationmessage to the receiver.6. The receiver's terminal initiatives a WAP connection and uses the WSPGET command to retrieve the handwritten message from the MMSC.7. Handwritten message is sent to the receiver's terminal as content ofa WSP GET RESPONSE command over the same WAP connection. The receiver'sterminal indicates, “Message received”8. The receiver's terminal acknowledges receipt with WSP POST commandover the same WAP connection.9. The MMSC uses WAP PUSH command to indicate to originators thatmessage was delivered. The originator's terminal indicates, “Messagedelivered”.

In FIG. 10, the interconnection of components of the present MMShandwritten messaging version is illustrated. The sender's wireless ordigital cellphone device 410 has a J2ME handwriting client installed forcapturing handwritten input as graphical data and sending it as an MMSmessage using the standard MMSC protocol. The MMS message is received ata mobile network center 420 which includes a WAP gateway, and MMS(message processing) center, a PushProxy gateway and SMSC transmissionprotocol. The MMS message from the sender is “pushed” to the recipient'swireless or digital cellphone device 430 which similarly has a J2MEhandwriting client installed for viewing (and sending back replies to)the MMS message. The recipient can thus view the handwritten message asa graphical image on the mobile receiving device.

Handwritten Messaging System for Mobile Devices

In FIG. 11, a handwritten messaging system for mobile devices isillustrated in which an Originator writes on a touch-sensitive displayscreen of a mobile device which has a handwriting messaging componentoperable with a messaging client on the mobile device for capturing thehandwritten message as graphical data and sending it as a handwrittenelectronic message on a data transmission network. The handwritingmessaging component is a small-footprint software module programmed forthe computing environment used in the computing platform of the mobiledevice. It operates by setting up a data capture area of the messagingclient as a handwriting or handdrawing Editor/Viewer on thetouch-sensitive display screen, such as shown for the PDA in FIGS. 7Aand 7B. The user can thus write a message directly on the touch displayscreen and click the “Send” icon of the messaging client to send ahandwritten electronic message.

The data transmission network can be wired or wireless type. Examples ofwireless data transmission include WiFi “hotspots” (nodes) connected tothe Internet with email and other types of data messaging handled byInternet servers, or local wireless networks with a local network serveror each mobile device acting as a local server to other mobile devicesin the locality. The local wireless network can use any suitable datatransmission technology, such as Bluetooth™ wireless point-to-pointtransmission or WiFi RF transmission. The handwritten electronicmessaging of the Originator is sent via the data transmission network tothe mobile device of a Receiver which also has a handwriting messagingcomponent for receiving and similarly sending handwritten or handdrawnmessages.

It is envisioned that the handwritten messaging system of the presentinvention is particularly useful for messaging via mobile game playerdevices. The use of handwriting and drawings has been a part of gameplaying since ancient times. Many popular games are based on handwritingand drawings such as Tic Tac Toe, Hangman, and Pictionary®. Otherpopular pastimes that involve handwriting include coloring pictures anddrawing together. The Handwriting Messaging System for Game Deviceswould allow people from all over the world to use handwriting anddrawings to play electronic version of these types of games with eachother.

The mobile game player device can be a portable or console-based gameplayer with a network connection (either wireless or wired), videodisplay and pad that allows handwriting input via a stylus. Thehandwriting system presents an area on the screen display for capturinghandwriting via the stylus, digitizes the stylus movements on the padand transmits the digitized data in an electronic messaging format toone or more player's devices. The transmission may be eitherpoint-to-point or through a network server connected to multiple clientdevices. For point-to-point connection, the handwriting client componentoperable on one mobile device can establish itself as a wireless hub andsend data by wireless transmission within its range to another clientcomponent operating on another mobile device. The handwriting clientcomponent can send the handwritten electronic message like email whichis placed in the receiving “mailbox” of the user, or as “push email”which is notified to the user, or in “instant messaging” format in whichthe system alerts the recipient that a message has been received and/orautomatically displays the handwritten message on the recipient'sdisplay screen. Alternatively, the system can be configured where themessage display area is used for handwritten “chat” in which one userwrites a message that automatically appears on the other user's screen,then the other user can markup the message or write a reply on ascrollable portion of display area, back and forth with each other.

It is understood that many other modifications and variations may bedevised given the above description of the principles of the invention.It is intended that all such modifications and variations be consideredas within the spirit and scope of this invention, as defined in thefollowing claims.

1. A handwritten messaging system operable on mobile devices connectedto a data transmission network comprising: (a) a server componentoperable on a server computer connected to a data transmission networkfor receiving a handwritten message in electronic format sent from auser of a mobile device and delivering the handwritten message in realtime to a mobile device of another recipient to whom it is addressed;(b) the mobile device having a handwriting messaging component operablewith a messaging client of the mobile device having a connection to thedata transmission network, wherein said handwriting messaging componentsets up a handwriting data capture area within said messaging clientinto which the user can enter handwritten input through a suitablemanual input device, and said handwriting data capture area operates tocapture the handwritten input as graphical data and send it as a messagein electronic format on the data transmission network; and (c) themobile device being connected to the data transmission network having amessaging client for sending and receiving the handwritten electronicmessage sent from the mobile device of the user to that of the recipientvia the network and for inputting it and then viewing it as ahandwritten message via the messaging clients of the respective devices.2. A handwritten messaging system according to claim 1, wherein thehandwriting messaging component is operable for also setting up agraphical data viewing area for viewing handwritten electronic messagessent to the user via the data transmission network.
 3. A handwrittenmessaging system according to claim 1, wherein the handwritten messagingcomponent is a small-footprint software module installed with themessaging client of the mobile device.
 4. A handwritten messaging systemaccording to claim 1, wherein the mobile device is a portable gameconsole or game player device.
 5. A handwritten messaging systemaccording to claim 1, wherein the mobile device is mobile game playerwith a wireless connection to a wireless data network.
 6. A handwrittenmessaging system according to claim 1, wherein the handwritten messagingcomponent of the mobile device converts the handwritten input to agraphical data attachment to an email message.
 7. A handwrittenmessaging system according to claim 1, wherein the handwritten messagingcomponent of the mobile device converts the handwritten input to “pushemail” format wherein the recipient is automatically alerted of receiptof a handwritten message on the recipient's mobile device.
 8. Ahandwritten messaging system operable on mobile devices connected to adata transmission network comprising: (a) a server component operable ona server computer connected to a data transmission network for receivinga handwritten message in electronic format sent from a user of a mobiledevice and delivering it to a mobile device of another recipient to whomit is addressed; (b) the mobile device having a handwriting messagingcomponent operable with a messaging client of the mobile device having aconnection to the data transmission network, wherein said handwritingmessaging component sets up a handwriting data capture area within saidmessaging client into which the user can enter handwritten input througha suitable manual input device, and said handwriting data capture areaoperates to capture the handwritten input as graphical data and send itas a message in electronic format on the data transmission network; and(c) the mobile device being connected to the data transmission networkhaving a messaging client for sending and receiving the handwrittenelectronic message sent from the mobile device of the user to that ofthe recipient via the network and for inputting it and then viewing itas a handwritten message via the messaging clients of the respectivedevices, wherein the handwritten messaging component of the mobiledevice converts the handwritten input to “instant messaging” formatwherein the handwritten message is automatically displayed on therecipient's mobile device.
 9. A handwritten messaging system operable onmobile devices connected to a data transmission network comprising: (a)a server component operable on a server computer connected to a datatransmission network for receiving a handwritten message in electronicformat sent from a user of a mobile device and delivering it to a mobiledevice of another recipient to whom it is addressed; (b) the mobiledevice having a handwriting messaging component operable with amessaging client of the mobile device having a connection to the datatransmission network, wherein said handwriting messaging component setsup a handwriting data capture area within said messaging client intowhich the user can enter handwritten input through a suitable manualinput device, and said handwriting data capture area operates to capturethe handwritten input as graphical data and send it as a message inelectronic format on the data transmission network; and (c) the mobiledevice being connected to the data transmission network having amessaging client for sending and receiving the handwritten electronicmessage sent from the mobile device of the user to that of the recipientvia the network and for inputting it and then viewing it as ahandwritten message via the messaging clients of the respective devices,wherein the handwritten messaging component of the mobile deviceconverts the handwritten input to “chat” format in which a handwrittenmessage written by one user is automatically displayed on the otheruser's screen, then the other user can markup the handwritten message orwrite a reply on a scrollable portion of data capture area, back andforth with each other.
 10. A handwritten messaging system according toclaim 9, wherein the handwritten messaging component of the mobiledevice converts the handwritten input to “chat” format in which eachuser can write a reply on a scrollable portion of the data capture area,back and forth with each other.
 11. A handwritten messaging method foruse with mobile devices comprising: (a) setting up a data capture areafor handwriting input in a display screen for a messaging client of amobile device of a user (as sender) connected to a data transmissionnetwork, and using a manual input device operatively coupled to thehandwriting data capture area to enable the sender to enter handwritteninput into the handwriting data capture area of the messaging client;(b) composing a handwritten message on the mobile device of the senderby entering handwritten manual input with the input device in thehandwriting data capture area of the messaging client, capturing thehandwritten input as graphical data, and sending it in real time as ahandwritten electronic message on the data transmission network to arecipient to whom the handwritten electronic message is addressed; and(c) receiving the handwritten electronic message sent from the senderthrough the data transmission network to a mobile device of therecipient to whom it is addressed, wherein the mobile devices areconnected to the data transmission network and enable inputting thehandwritten message in the mobile device of the user and viewing it as ahandwritten message in the mobile device of the recipient via themessaging clients of the respective devices.
 12. A handwritten messagingmethod according to claim 11 further comprising setting up a graphicaldata viewing area for viewing handwritten electronic messages sent tothe mobile device of the user via the data transmission network.
 13. Ahandwritten messaging method according to claim 11, wherein the datacapture area is set up by a small-footprint software module installedwith the messaging client of the mobile device.
 14. A handwrittenmessaging method according to claim 11, wherein the mobile device is aportable game console or game player device.
 15. A handwritten messagingmethod according to claim 11, wherein the mobile device is mobile gameplayer with a wireless connection to a wireless data network.
 16. Ahandwritten messaging method according to claim 11, wherein themessaging client of the mobile device converts the handwritten input toa graphical data attachment to an email message.
 17. A handwrittenmessaging method according to claim 11, wherein the messaging client ofthe mobile device converts the handwritten input to “push email” formatwherein the recipient is automatically alerted of receipt of ahandwritten message on the recipient's mobile device.
 18. A handwrittenmessaging method for use with mobile devices comprising: (a) setting upa data capture area for handwriting input in a display screen for amessaging client of a mobile device of a user (as sender) connected to adata transmission network, and using a manual input device operativelycoupled to the handwriting data capture area to enable the sender toenter handwritten input into the handwriting data capture area of themessaging client; (b) composing a handwritten message on the mobiledevice of the sender by entering handwritten manual input with the inputdevice in the handwriting data capture area of the messaging client,capturing the handwritten input as graphical data, and sending it as ahandwritten electronic message on the data transmission network; and (c)receiving the handwritten electronic message sent from the senderthrough the data transmission network to a mobile device of a recipientto whom it is addressed wherein the mobile devices are connected to thedata transmission network and enable inputting the handwritten messagein the mobile device of the user and viewing it as a handwritten messagein the mobile device of the recipient via the messaging clients of therespective devices, and the messaging client of the mobile deviceconverts the handwritten input to “instant messaging” format wherein thehandwritten message is automatically displayed on the recipient's mobiledevice.
 19. A handwritten messaging method for use with mobile devicescomprising: (a) setting up a data capture area for handwriting input ina display screen for a messaging client of a mobile device of a user (assender) connected to a data transmission network, and using a manualinput device operatively coupled to the handwriting data capture area toenable the sender to enter handwritten input into the handwriting datacapture area of the messaging client; (b) composing a handwrittenmessage on the mobile device of the sender by entering handwrittenmanual input with the input device in the handwriting data capture areaof the messaging client, capturing the handwritten input as graphicaldata, and sending it as a handwritten electronic message on the datatransmission network; and (c) receiving the handwritten electronicmessage sent from the sender through the data transmission network to amobile device of a recipient to whom it is addressed, wherein the mobiledevices are connected to the data transmission network and enableinputting the handwritten message in the mobile device of the user andviewing it as a handwritten message in the mobile device of therecipient via the messaging clients of the respective devices, and themessaging client of the mobile device converts the handwritten input to“chat” format in which a handwritten message written by one user isautomatically displayed on the other user's screen, then the other usercan markup the handwritten message or write a reply on a scrollableportion of data capture area, back and forth with each other.
 20. Ahandwritten messaging method according to claim 19, wherein themessaging client of the mobile device converts the handwritten input to“chat” format in which each user can write a reply on a scrollableportion of the data capture area, back and forth with each other.